Information handling devices (“devices”), for example smart phones, tablet devices, laptop computers, and the like, are capable of receiving and analyzing voice input data. For example, a device can receive the voice input data when a user is providing dictation to the device, conversing with another user using the device, providing commands to the device or an application on the device, and the like. The device can use the voice input data to determine various characteristics (“voice biometrics”) associated with a user's voice. Advances in technology have enabled users to leverage this voice biometric data to develop speaker recognition methodologies. These methods may be used to determine the identity of an unknown speaker.